Skip to content

Fix K8s executor pod_override stringified without cncf provider#67895

Open
ephraimbuddy wants to merge 1 commit into
apache:mainfrom
astronomer:afl-32-decouple-v1pod-serialization
Open

Fix K8s executor pod_override stringified without cncf provider#67895
ephraimbuddy wants to merge 1 commit into
apache:mainfrom
astronomer:afl-32-decouple-v1pod-serialization

Conversation

@ephraimbuddy
Copy link
Copy Markdown
Contributor

Airflow 3 deployments that run the Kubernetes executor with the kubernetes
package installed but without apache-airflow-providers-cncf-kubernetes
had their executor_config pod_override V1Pod silently converted to a
string during Dag serialization. The stringified pod was persisted to
serialized_dag._data and task_instance.executor_config, so affected tasks
stayed in queued and never ran. (Airflow 2 handled this case via an in-core
PodGenerator fallback that was removed ahead of the 3.0 release.)

Root cause

_has_kubernetes() in serialized_objects.py hardcoded an import of the cncf
provider's PodGenerator for V1Pod ser/deser. When that provider isn't
installed the import fails, _has_kubernetes() caches False, the V1Pod
serialization branch short-circuits, and default_serialization() falls back
to str(var).

Fix

Serialize/deserialize V1Pod objects directly through
kubernetes.client.ApiClient — which is all PodGenerator.serialize_pod and
PodGenerator.deserialize_model_dict wrap anyway — so airflow-core no longer
depends on any specific Kubernetes-executor provider. V1Pod ser/deser now
works with the cncf provider, an alternative executor package, or just the bare
kubernetes package installed. The backcompat unpickle path in
ensure_pod_is_valid_after_unpickling is decoupled the same way.

Tests

Added a regression test that makes apache-airflow-providers-cncf-kubernetes
unimportable and asserts V1Pod still round-trips through
BaseSerialization. The test blocks the ...pod_generator submodule directly
(not just the parent package) so it can't be fooled by import ordering when the
module is already cached by a sibling test. Existing K8s serialization tests
(test_serialize_v1pod_attempts_import_before_serializing, the
executor_config_pod fixtures, TestExecutorConfigType) continue to pass —
serialization output is byte-identical to the previous cncf-based path.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

Airflow 3 deployments using the Kubernetes executor with the `kubernetes`
package installed but without `apache-airflow-providers-cncf-kubernetes`
had their `executor_config` `pod_override` V1Pod silently converted to a
string during Dag serialization. The stringified pod was persisted to
`serialized_dag` and `task_instance.executor_config`, so affected tasks
stayed in `queued` and never ran.

`_has_kubernetes()` hardcoded an import of the cncf provider's
`PodGenerator` for V1Pod ser/deser. When only an alternative executor
package is installed, that import failed, `_has_kubernetes()` cached
False, and serialization fell back to `str(pod)`. Airflow 2 had an
in-core `PodGenerator` fallback for this case that was removed ahead of
the 3.0 release.

Serialize and deserialize V1Pod objects directly through
`kubernetes.client.ApiClient`, which is all `PodGenerator.serialize_pod`
and `deserialize_model_dict` wrap anyway. V1Pod ser/deser now works with
the cncf provider, an alternative executor package, or the bare
`kubernetes` package installed. The backcompat unpickle path in
`ensure_pod_is_valid_after_unpickling` is decoupled the same way.
@ephraimbuddy ephraimbuddy added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label Jun 2, 2026
@ephraimbuddy ephraimbuddy added this to the Airflow 3.2.3 milestone Jun 2, 2026
@ephraimbuddy ephraimbuddy self-assigned this Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:DAG-processing backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants